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'METHOD AND SYSTEM FOR AUTOMATICALLY CONTROLLING A PERSONALIZED 

NETWORKED ENVIRONMENT 

FIELD OF THE INVENTION 
The invention relates to a method and system for using face 
recognition to initiate networked appliances associated with an 
individual when they enter an environment. 

BACKGROUND OF THE INVENTION 
Current environments, such as homes, contain many 
appliances ranging from illumination sources (i.e., lamps) and 
climate control systems (i.e., air conditioners and heating 
systems) to entertainment devices (i.e., stereo, television) and 
message delivery systems (i.e., voicemail, fax, email). When an 
individual enters an environment, they must affirmatively 
perform an action to initiate each appliance and adjust each 
appliance to suit their tastes or deliver their personalized 
information. For example, to check one's voicemail, a user must 
dial up their voicemail carrier, enter their mailbox and pin 
number (for location and identification purposes) and perform a 
series of keystrokes to finally obtain their messages. 
Additionally, a user must sort through various messages that may 
be directed to other individuals before finding and receiving 
their own, such as with a telephone answering machine. 
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Face recognition identification technology, as described in 
U.S. Patents 6,111,517 and 6,108,437, herein incorporated by 
reference, has been employed to identify individuals and 
restrict access to controlled environments. Present face 
recognition methods used in person identification systems, 
typically employ a face recognizer which determines the identity 
of a human face image. In order to identify the image, the 
system compares the face to other faces, or performs a feature 
extraction and compares features to those recorded in a 
database . 



SUMMARY OF THE INVENTION 
According to the invention, appliances can be automatically 
initiated and/or adjusted as a user enters an environment, such 
as a home or a room. The environment is networked such that 
there is a communication link between a camera monitoring the 
entrance to the environment, a face recognition system, and the 
appliances . 

According to one aspect of the invention, a method of 
automatically controlling a networked environment includes a 
setup step and an initiating step. The setup step includes 
capturing a face image of an individual, storing the face image 
(i.e., in a memory), and associating at least one appliance 
setup to the face image. 
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The initiating step includes comparing the face' image to stored 
face images, retrieving the at least one appliance setup for a 
matched face image, and initiating the at least one appliance 
setup. An appliance setup controls turning on or off an 
appliance, such as an illumination source, a television, or a 
stereo. Not only can the appliance be turned on or off, but the 
user can enter a quantifiable term associated with the appliance 
such as a level, a temperature, a volume, or a channel, to which 
the appliance will be set upon initiation. 

According to an embodiment of the invention, when several 
users frequent the same environment over a course of time, the 
method prioritizes each user on a hierarchy of control to 
determine which user's setting to initiate. The prioritization 
includes associating a matched face image with a hierarchy of 
control over the at least one appliance setup. This is 
accomplished by first determining if a second individual 
previously initiated the at least one appliance setup, and if 
so, determining the hierarchy of control over the at least one 
appliance setup associated with the second individual. The 
method further includes, initiating the at least one appliance 
setup associated with the matched face image only if no second 
individual previously initiated the at least one appliance 
setup, or if the matched face image ranks higher within the 
hierarchy of control. If users have the same rank, or neither 

3 

F:\WPDOCS\WX\US0 1011 3 .SPEC.doc 




usoion3 



one has a rank, a default setting could select between them or 
an average setting may be initiated (i.e., an average of the 
light levels each user selected) . 

According to another embodiment a password can be entered 
by the individual which, is verified as matching a preset 
password before storing the facial representation of the 
individual . 

According to several embodiments of the invention, the at 
least one appliance setup includes a message storage device 
which receives and records incoming messages, determines if a 
message is associated with the individual, and recalls and 
recites the message associated with the individual upon the 
initiation step. The messages can be, but are not limited to, 
telephonic messages recorded on an answering machine or 
voicemail, email messages, facsimile messages, video clips, 
text-embossed images, or some combination thereof. 

According to a second aspect of the invention, a system for 
automatically controlling a personalized networked environment 
upon an individual entering an environment includes a face 
recognition system for storing a face image of the individual, 
and determining the identity of the individual upon their 
entering the environment. .The system also includes a memory for 
storing the face image and at least one appliance setup, at 
least one appliance setup capable of being initiated by the 
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system, and a network connecting the appliance setup to the 
system. 

BRIEF DESCRIPTION OF THE DRAWING 
Fig. 1 is a block diagram of the networked system of the 

inventions- 
Fig. 2 is a flow chart depicting the storage steps of the 

method which occur prior to entering the environments- 
Fig. 3 is a flow chart depicting the steps of the method 

which occur upon a user entering the environment; 

Fig. 4 is a flow chart depicting the steps of the method 
which occur when more than one user occupies the same 
environment . 

DETAILED DESCRIPTION OF THE PRIMARY EMBODIMENT 
Referring to Fig. 1, networked system 19 uses face 
recognition to initiate networked appliances associated with an 
individual when they enter an environment. Initially, face 
recognition system 10 performs a registration of a user's face 
prior to the user leaving their house. Detector 4 detects the 
presence of a user and camera 1 captures their face and stores 
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it in- memory 2 using video face capture techniques well known in 
the art. The user may be required to enter an access code or 
some form of authorization before being permitted to register. 
The user also enters networked appliances he or she wishes to be 
initiated upon their return into CPU 3. Memory 2 stores these 
face captures and networked appliances. 

Upon returning to their house, detector 4 detects the 
presence of a person by, for example, checking every 5 seconds 
for changes in the environment, or checking when a door to the 
environment is opened. Comparison/recognition system 5 compares 
the user' s face to those stored in memory 2 and, when a match is 
found, recognizes the user according to face recognition methods 
known in the art. Once the user is recognized, determining 
means 12 determines if any recorded messages correspond to the 
user. The user is told if and how many messages they have and 
any messages that correspond to the user are then recited by 
recitation means 13. Networked system 19 also initiates 
appliance 16 after acquisition requesting/receiving means 14 
recalls from memory 2 the appliance 16 with which the user 
requested to be associated. 

Fig. 2 depicts the method steps of the system of Fig. 1 
prior to a user leaving their house. Face recognition system 10 
performs face capture step 20 via camera 1. Memory 2 stores the 
face capture according to storage techniques known in the art in 
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step 21. In step 22, CPU 13 requests that the user input 
association information. For example, in steps 25-27 CPU 13 
requests that the user input their name, an appliance with which 
they wish to be associated, and a personal identification number 
(pin) and/or voiceprint by which they can access the memory. 
The examples listed in steps 25-27 represent a non-exhaustive 
list of the association information the method can request. For 
purposes of this example, the user enters "Joe Smith'' for their 
name, a light source and an air conditioner as appliances, and a 
pin of ''1234''. The user enters the information by, for example, 
typing on a keyboard or speaking into a voice recognition 
device. The system can also be configured to perform 
association automatically, for instance, by making a telephone 
answering machine associated with the user. Memory 2 saves the 
association information in step 23, and CPU 13 associates the 
association information with the stored face capture image in 
step 24. With respect to step 26, CPU 13 may also request a 
user to input a quantitative term associated with the appliance, 
such as a brightness setting for the light source, or a 
temperature associated to a heating source. 

Once the steps of Fig. 2 are completed a user can then 
leave their house and upon their return, the steps listed in 
Fig. 3 will occur. 
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Referring to Fig. 3, upon entry through the doorway to 
their house, detector 4 detects their presence and camera 1 
captures the user's face, as in step 20 of Fig. 1. 
Comparison/recognition system 5 compares the face capture with 
all previous stored captures, or performs a feature extraction 
which compares features of the user's face to those recorded in 
a database. Comparison/recognition system 5 determines the 
user's identity in step 34. CPU 3 recalls inputted association 
information stored in memory 2 in step 35 and may address the 
user through speech means (i.e., reciting ''Welcome home, Joe 
Smith") . In a particular embodiment of the invention, 
determining means 12 determines which messages are associated 
with the user in step 36. If message recording device 11 is an 
answering machine this determination can occur, for example, by 
selecting those messages which include the user' s name 
(previously entered or recorded in step 12) . If the message 
recording device is, for example, a voicemail system, the user's 
personal voicemail box can be accessed. Message 
receiving/recording device 11 recalls any messages associated 
with the user in step 38 and recitation means 13 automatically 
recites those messages to the user in step 39. Recitation means 
11 may also recite the number of messages determining means 12 
determined in step 36 to be associated with the user. 
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'Either simultaneously or nearly simultaneously to steps 36 
and 38-39, CPU 3 initiates the light source and air conditioner 
which the user entered in step 22, Further, if the user entered 
a quantified term to which they wished the appliance set (i.e., 
the brightness of the light, or the temperature of the air 
conditioner) , CPU 3 initiates the appliance to that amount in 
step 37. 

Another possible embodiment of the invention provides that 
after the face capture in step 20, CPU 3 requests that the user 
provides their pin number in step 31. The user enters their pin 
number, CPU 3 receives it in step 32, and verifies that the pin 
number ''1234" matches the user's face capture in step 33. If 
the pin number or voiceprint does not match or is not verified, 
CPU 3 repeats the request for input of a pin number. If no 
verification occurs after several attempts, the method may 
perform an additional face capture, or simply maintain its 
current settings. 

Control problems may arise when a user enters the house and 
a second user with second control settings is already in the 
house. To remedy such situations Fig. 4 depicts the steps 41- 
45. After the user enters the house, camera step 10 captures 
their face and comparison/recognition system 5 performs a 
recognition at step 41. CPU 3 associates the user with the 
light source and air conditioner in step 35. CPU 3 also 
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associates the user with a control hierarchy in step 42, whereby 
the user is ranked for each appliance selected with regard to 
whose preference, of all registered users, controls. CPU 3 
determines if a second user has previously initiated the light 
source and/or the air conditioner and if that user is still 
present in step 43. If no second user is detected, CPU 3 
initiates the light source and air conditioner according to the 
user's preference in step 45. If, however, a second user is 
detected and is present, CPU 3 determines the previous user's 
control rank with respect to the user's in step 44. If the 
second user has a lower rank, CPU 3 initiates the light source 
and air condition according to the user's preference in step 45. 
If the second user has a higher rank, CPU 3 instead maintains 
the current settings of the appliance (s) in step 46. If both the 
first and second users have the same rank, CPU 3 initiates a 
default setting or an average setting in step 47. 

The preceding expressions and examples are exemplary and 
are not intended to limit the scope of the claims that follow. 
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